Skip to content

release: GeoBrix 0.5.0 — virtual tiles, v2 tile struct, CRS families - #72

Draft
mjohns-databricks wants to merge 828 commits into
mainfrom
beta/0.5.0
Draft

release: GeoBrix 0.5.0 — virtual tiles, v2 tile struct, CRS families#72
mjohns-databricks wants to merge 828 commits into
mainfrom
beta/0.5.0

Conversation

@mjohns-databricks

Copy link
Copy Markdown
Collaborator

GeoBrix 0.5.0 — release PR (beta/0.5.0main)

Promotes the 0.5.0 line to main (~496 commits). Versions are bumped (pom.xml + __version__ = 0.5.0). Full change list: docs/docs/release-notes.mdx → "What's new in v0.5.0".

Highlights

  • Virtual tiles — bytes-free windowed reads for large rasters. Lightweight cog_gbx/raster_gbx/gtiff_gbx readers can emit virtual tiles (source path + pixel window, no bytes); pixels are read lazily one window at a time. A virtual row is ~100 B vs 148–527 KB materialized (~1,400–5,000× smaller), dissolving Serverless OOM on large-raster fan-out. Lightweight raster readers now default to virtual (breaking behavior change).
  • v2 8-field tile structcellid, raster (nullable), path, window, clip_polygon, clip_crs, crs, metadata. Both tiers read v1 and v2 and always emit v2.
  • Every lightweight rst_* is virtual-tile-aware via one shared open path (header-only accessors; reference/passthrough ops stay virtual; pixel ops materialize only their window). New force-output params: virtualize_dir / virtualize_prefix / materialize.
  • COG preparation lane — new file_gbx (path lister) + cog_gbx writer (master-COG prep) + cog_gbx reader (COG-aware windowed read: tileSize/overlapPercent/clipPolygons/windows). Reader splitStrategy now defaults to none.
  • CRS families (both tiers)gbx_st_{crs,setcrs,transformcrs} (VectorX) and gbx_rst_{crs,setcrs,transformcrs} (RasterX): authority-string CRS handling that survives non-EPSG round-trips.
  • Plus per-function 4-tab docs examples, Scala-derived DESCRIBE FUNCTION signatures, and many fixes.

Pre-merge gate (requested)

  • Complete the eo-series' use of virtual tiles (lightweight tier / Serverless): notebooks showcase virtual tiles (narrative + functions used), the eo-series doc page (docs/docs/notebooks/eo-series.mdx) and the eo-series README.md reflect the support, validated on Serverless.

Opened as draft until the gate above is met. beta/0.5.0 CI is the dev-branch signal (not a merge gate); the main build runs on merge.

This pull request and its description were written by Isaac.

mjohns-databricks added a commit that referenced this pull request Aug 13, 2026
PR #72's build failed on `black --check`. CI lints as `isort && black && flake8
src test`, so black's failure short-circuited before flake8 — hiding pre-existing
flake8 debt that surfaces once black passes. This clears both in one pass:

- black-reformat 7 drifted files (bench/{runner,spec}.py, test/bench/{cluster,
  results,spec}, test/pyrx/{core_agg,v2_tile_output_invariant}).
- flake8: drop 3 unused imports (F401 in pyrx/_serde.py, test/rasterx/
  test_udtf_error_row.py, test/vectorx/test_crs.py) and mark the bench helper
  run_spark_path with # noqa: C901 (matches the _render precedent).

No functional change; none of these files were part of the vizx/eo-series work —
this is latent tree-wide lint debt that was masked by the black short-circuit.

Co-authored-by: Isaac
…nvention

Two additions before authoring, per user emphasis that VectorX is geometry-dominated: (1) a uniform output-representation convention (elision ... + format annotation for [E]WKB/MVT binary, [E]WKT, GeoJSON, CRS strings; ~60-char cell cap) stated once in the T1 Conventions section and enforced by the docs-examples guard; (2) an explicit per-function review gate that all four tabs use the SAME example (same fixture/op/args) so output renders consistently, with genuine tier differences labeled, never fabricated or hidden.

Co-authored-by: Isaac
…extension

Adds the VectorX tabbed-docs foundation layer that T2-T5 build on:

- Extend generate-function-info.py _TIER_SCANS: widen light-Python glob
  from rasterx_*_python_light.py to *_python_light.py (picks up the new
  vectorx light file), and add vectorx_functions.py as a second python-heavy
  scan path. RasterX bindings unchanged (spot-checked rst_avg/rst_width).

- Add VectorX fixture loaders to _fixtures.py: tin_df/tin_df_heavy (4
  WKB POINT Z, non-flat Z range 10 m), mvt_features_df/..._heavy (2 tile-
  local WKB POINTs), geom_ewkt_df/..._heavy (SRID=4326;POINT(13 42)), and
  legacy_geom_df/..._heavy (Mosaic InternalGeometry struct). Adds setup
  view builders: create_setup_views_vectorx_light/heavy for the four views
  (tin_survey, mvt_features, vector_geoms, legacy_geoms).

- Add 8 new fixture tests in test_fixtures_helpers.py asserting non-degenerate
  data (rows>0, WKB size, Z range, EWKT has SRID=, struct typeId=1, all 4
  views accessible after setup).

- Create vectorx_functions_python_light.py scaffold: module docstring,
  fixture helpers, setup example; per-function examples deferred to T2-T5.

- Create test_vectorx_functions_python_light.py: 4 smoke tests (import,
  setup exists, setup executes, views created). Autouse fixture registers
  pyvx + creates the four VectorX views.

- Regenerate function-info.json via gbx:docs:function-info; 180 entries,
  VectorX bindings remain ["sql"] (light/heavy examples not yet authored).

- Add Setup + Conventions section to vectorx-functions.mdx (after tier-
  availability table, before Vector tile output): placeholder->view->backs
  table, 4-tab reading guide, uniform output-representation convention
  ([E]WKB/MVT/WKT/CRS ~60-char cap), CRS-family-returns-BINARY note,
  light-UDTF LATERAL form convention. Import FunctionExamples component +
  vectorxLightCode + scalaApiExamplesCode for T2-T5 use.

Test: 1100 passed, 1 deselected, 0 failed (baseline 1088+12 new).
Docs build: gbx:docs:build --no-restart ✓.

Co-authored-by: Isaac
…t_transformcrs)

Adds 4-tab (SQL / Python light / Python heavy / Scala) examples for the three
CRS-family VectorX functions and wires them into FunctionExamples blocks on the
vectorx-functions doc page.

Changes:
- vectorx_functions_python_light.py: 3 light examples + outputs (pyvx tier, uses
  vector_geoms setup view, POINT(13,42) SRID=4326 fixture)
- vectorx_functions.py: 3 heavy examples + outputs (vectorx tier, same fixture)
- ScalaApiExamples.scala: 3 Scala examples + outputs
- test_vectorx_functions_python_light.py: 3 per-function light tests asserting
  real values (EPSG:4326 string, non-null EWKB bytes)
- test_vectorx_functions.py: vectorx_heavy_setup fixture + 3 heavy tests
- vectorx-functions.mdx: CodeFromTest → FunctionExamples for st_crs / st_setcrs /
  st_transformcrs (bindings=[sql, python-light, python-heavy, scala])
- test_example_output_tables.py: extend _example_modules() to scan vectorx files
- function-info.json: regenerated; all 3 CRS functions now carry all 4 bindings

Co-authored-by: Isaac
C1: st_setcrs and st_transformcrs SQL tabs wrapped the function in
gbx_st_crs(...), showing STRING instead of the BINARY the functions
actually return. Rewritten to direct invocations showing the real
BINARY output with (EWKB binary — ...) annotation.

C2: All 3 CRS SQL examples used inline literals and multi-column
showcases differing from the Python/Scala tabs (POINT(11,42) vs 13,42;
multi-col vs single-col). Aligned to the shared vector_geoms fixture
(SRID=4326;POINT (13 42)) and single-column form matching all other tabs.

Also: added test_st_setcrs_stamps_different_crs (SQL + light) that
stamps EPSG:3857 on an SRID=4326 geom and round-trips through st_crs
to verify the SRID actually changed — prev test stamped 4326 onto 4326,
a silent no-op. Updated vectorx_registered fixture to create vector_geoms
view so SQL examples execute. Regenerated function-info.json.

Co-authored-by: Isaac
…ramid)

Add 4-tab FunctionExamples blocks for both MVT functions, converting from the
prior single-tier CodeFromTest to the full SQL/Python-light/Python-heavy/Scala
standard. Fixtures reuse the `mvt_features` view (tile-local WKB POINTs) for
st_asmvt and inline WGS-84 POINT(0,0) for st_asmvt_pyramid.

Fix a correctness bug in the stalled implementation: the heavy Python and Scala
examples incorrectly referenced `t.tile.z` / `t.tile.mvt_bytes` for the
DataFrame API struct path. The generator `.alias("t")` yields direct fields
`t.z`, `t.x`, `t.y`, `t.mvt_bytes` (no `tile` wrapper); the `tile` wrapper
only appears in SQL `LATERAL VIEW ... AS tile` syntax. Switched to
`.selectExpr("t.z AS z", ...)` and `rows[0]["mvt_bytes"]` to match the
confirmed unit-test pattern in `test_st_asmvt_pyramid.py`.

Also add the missing `st_asmvt_sql_example_output` and
`st_asmvt_pyramid_sql_example_output` constants (eliminating build warnings)
and regenerate function-info.json so both functions gain all 4 tier bindings.

Doc tests: 1126 passed, 0 failed (baseline 1088; +38 from T1–T3).
Docs guard: OK (431 output constants, 511 tabs, 124 annotation groups).
Build: clean (0 warnings).

Co-authored-by: Isaac
…expression

Chaining Column-API tile functions in one expression (rx.rst_slope(rx.rst_setcrs(...))) fuses into a single Python worker, so intermediate tiles pass in-worker without a JVM round-trip — the per-batch boundary is crossed once for the whole chain, not once per step. Materializing intermediates across separate select/withColumn steps re-serializes at each stage. Composes with virtual-tile deferral. (Insight surfaced from the FILE Python-UDF execution design; the note is about current chaining behavior, not FILE.)

Co-authored-by: Isaac
…cation

Three correctness fixes on the T3 MVT tabbed examples:

1. SQL examples returned `length(gbx_st_asmvt(...))` (INTEGER) instead of
   the raw BINARY, violating "the shown code invokes THE NAMED FUNCTION and
   nothing wraps it to change the result type". Dropped the length() wrapper;
   both SQL examples now return the BINARY column directly (matching the
   `... (MVT binary)` annotation). SQL tests updated to assert non-null/non-empty
   bytes instead of a byte-count integer.

2. `st_asmvt_pyramid_python_light_example` returned `rows[0]["z"]` (an int),
   leaving the MVT encoding path unverified on the light tier. Changed to return
   `rows[0]["mvt_bytes"]`; test updated to assert `isinstance(result,(bytes,
   bytearray)) and len(result) > 0`, mirroring the heavy tier test.

3. Pyramid output annotation was split across tiers (SQL/light used "one row per
   intersecting tile across zoom levels 0–2"; heavy/Scala used "one row per tile;
   use t.z …"). The annotation consistency guard requires identical trailing
   annotations on identical tables; unified all four tabs to the SQL/light phrasing.

Scala minor: `result.count()` → `result.show()` so the shown code matches the
displayed table output.

Doc tests: 1128 passed, 0 failed. Guard: OK (126 annotation groups). Build: clean.

Co-authored-by: Isaac
…n scoping

Doc-tests under api/ can only run as the whole dir (the _fixtures sibling import needs whole-dir collection), so there was no way to run just the changed functions. Add --filter '<expr>' which passes pytest -k while still collecting the whole path, so tasks touching a few functions run only those tests (e.g. --path api/ --filter 'st_triangulate or st_interpolate') instead of the full ~1128-test api suite each iteration. The host-only gbx:test:docs-examples guard remains the cross-tab consistency net.

Co-authored-by: Isaac
Adds per-function SQL/Python-light/Python-heavy/Scala examples for
st_triangulate, st_interpolateelevationbbox, st_interpolateelevationgeom.
Converts three CodeFromTest blocks in vectorx-functions.mdx to
FunctionExamples and removes the now-redundant hand-written PySpark block.

Key implementation findings surfaced during authoring:
- TIN Scala expressions extend CollectionGenerator, not UDTF. SQL tab uses
  LATERAL VIEW (Hive generator syntax); light Python tab uses SQL standard
  LATERAL (pyvx UDTF). This matches the st_asmvt_pyramid pattern already
  in the docs.
- Single-field elementSchema (STRUCT<triangle BINARY>) unwraps the field
  type directly in the Python Column API: .alias("triangle") not
  .alias("t").select("t.triangle"), which fails with INVALID_EXTRACT_BASE.
- Spark SQL decimal literals (0.01) fail readDouble in the Scala expressions;
  integer literals (0) are used for tolerances (accepted as Int→Double; same
  behavior on the clean 4-point fixture).

All 3 functions now carry all 4 bindings in function-info.json.
Full api/ doc-test suite: 1143 passed (+15 vs baseline), 0 failed.

Co-authored-by: Isaac
Remove the outdated "_should_stream docstring" note that called
OpenFileGDB memory-bounding "a separate follow-up". The bounding
was already in place via _write_local_osgeo_gdb, which streams
fragments one at a time with OGR transaction batching. Update the
docstring to accurately describe the current implementation.

Add test_vector_gdb_streaming.py with two tests:
- test_write_local_osgeo_gdb_reads_one_fragment_at_a_time: unit-level
  spy via WeakRef-tracked feather.read_table calls, asserts max
  concurrent live fragment tables <= 1 (GAP-4 streaming invariant).
- test_gdb_multi_fragment_roundtrip_rows_geoms_crs: end-to-end
  multi-partition FileGDB write+read verifying row count, attribute
  values, geometry types, and CRS round-trip fidelity.

Co-authored-by: Isaac
…self-report

Add a static regression guard (test_materialize_regression_guard.py) that
asserts each of the four materializing entry points (file_ref_arg,
RasterGbxWriter.write, _fromfile_impl, CogGbxWriter.write) still references
the shared size-safety gate (_connect_aware_lru_sizing / materialize_decision).
A future refactor that drops the gate call and substitutes a raw unbounded
.read() will fail this test immediately, before reaching Serverless.

Add report_detected_cap(spark=None) -> int to ds/file_gbx.py (__all__) so
the T8 on-cluster validation can assert that the Serverless Connect session
detects the 64 MiB cap (the linchpin — a mis-detect silently applies the
256 MiB classic cap, which is the OOM vector on Serverless).

Add 5 unit tests for report_detected_cap in test_materialize_decision.py,
including a consistency check proving report_detected_cap and materialize_decision
use the exact same threshold boundary.

Add one CLAUDE.md bullet in the GDAL resource management section codifying the
Serverless-safe materialize policy as a REQUIRED project convention.

Co-authored-by: Isaac
…ize behavior

Add serverless-and-memory.mdx as the authoritative reference for connect-aware
memory caps (64 MiB Serverless/Connect, 256 MiB classic, GBX_STREAM_MAX_BYTES),
materialize-vs-virtual safety, FILE Delta-table fast path (~1.8–2 s / ~16–17×
faster than directory scan on Serverless), memory-safe write/ingest guarantees,
and explicit caveats for rst_fromfile(materialize=True), rst_fromcontent, and
large-tile write with pending warp/clip.

Correct the reader Serverless matrix: update directory scan timing from ~74 s
to ~30 s / 1k tiles; update the ~41× speedup claim to ~16–17×; reframe the
FUSE-of-FILE path as a memory-safety mechanism (not a speedup) in the Read
resolver section; add connect-aware cap documentation throughout.

Add :::caution admonitions to rst_fromfile(materialize=True) and rst_fromcontent
in raster-functions.mdx. Wire the new page into sidebars.js.

Co-authored-by: Isaac
The write-side materialize gates resolved the connect-aware stream cap by
inspecting a live SparkSession, but DataSource V2 write(iterator) is pickled
and runs per-partition on workers. On a Serverless worker there is no session,
so the cap silently fell back to 256 MiB classic instead of 64 MiB and a
64-256 MiB tile was mis-gated "stream" -> full materialize -> OOM.

Fix 1 (HIGH): materialize_decision gains cap_bytes, used directly when given
(skips session resolution). RasterGbxWriter and CogGbxWriter capture the cap in
__init__ (driver, via SparkSession.getActiveSession()) into self._cap and pass
cap_bytes=self._cap in write(). rst_fromfile bakes the driver cap as an f.lit
and _fromfile_impl (which runs inside a worker UDF) consumes it via cap_bytes;
the read gate file_ref_arg already baked the cap at plan-build (unchanged).

Fix 2 (MEDIUM): the "stream" branch decodes the whole raster into RAM, which
can be ~10x the compressed on-disk size. RasterGbxWriter.write and
rst_fromfile(materialize=True) now gate on the DECODED size read from the tile
header, not the compressed size. The read gate stays on compressed size (it
holds only raw bytes, decoding one window).

Fix 3 (MEDIUM, doc): large tile + pending warp/clip still falls back to full
materialize (accepted limitation; WarpedVRT streaming is backlog). Added one
test documenting the fall-back path is taken.

Fix 4 (LOW): empirically confirmed _windowed_materialize_bytes and
materialize_to_bytes emit identical tiled/blockxsize/blockysize/interleave, so
those keys were added to the profile-equivalence assertion (strengthened).

Fix 5 (LOW): softened the regression-guard docstring — it is a textual tripwire
that the gate call is not deleted, not proof of runtime Serverless safety.

Verified: 100 passed across the six affected suites in Docker; python lint clean.

Co-authored-by: Isaac
… (parity with read gate)

RasterGbxWriter.__init__ and CogGbxWriter.__init__ called bare
SparkSession.getActiveSession() to bake the connect-aware materialize cap.
In some DBR 14+/Connect threading contexts getActiveSession() returns None,
leaving both writers with the 256 MiB classic fallback — a Serverless mis-gate.

All read-gate surfaces (file_ref_arg, rst_fromfile) already use
_resolve_session_for_cap() (getActiveSession → getOrCreate fallback), so this
closes the asymmetry: all four cap-capture sites now resolve the session the
same robust way.

Co-authored-by: Isaac
Introduces MosaicOptions dataclass + parse_mosaic_options() in
cog_writer.py: parse and validate mosaic-mode options before any I/O.

- Mosaic mode triggers when `mosaic` is truthy or `gridSystem` is
  supplied; absent / mosaic=false → unchanged single-COG path.
- Phase A: only gridSystem='none' (native tiling) accepted; quadbin/
  bng/h3 raise ValueError with a clear, user-facing message.
- Contradictory combinations rejected at plan time: driverMode + mosaic,
  tileSize/overlapPercent + DGGS, gridMin/Max/StepResolution + none.
- Per-tile encoding options (compress/blockSize/predictor) pass through.
- CogGbxWriter.__init__ gains mosaic_opts= (None = single-COG, no
  behaviour change); cog.py calls parse_mosaic_options upfront so
  Tasks 2-3 can read self.mosaic_opts without re-parsing.
- 29 pure-Python tests; no regression in 3 existing cog_writer suites.

Co-authored-by: Isaac
Implements CogGbxWriter._write_mosaic(): when mosaic_opts is set, each
source is opened once and tiled into bounded, non-overlapping mini-COGs
via rasterio windowed reads.  Each window is memory-bounded by the tile
grid (decoded_size <= tile_size^2 × bands × itemsize; block-streaming
fallback via _windowed_materialize_bytes for the rare over-cap case).

New module-level symbols:
  _tile_grid_windows(w, h, tile_size, overlap_pct) — row-major Window
  generator with optional halo expansion and source-bounds clamping.
  _is_all_nodata(data, nodata) — pruneEmpty helper (NaN-aware).
  _MOSAIC_DEFAULT_TILE_SIZE = 1024.

write() branches on self.mosaic_opts first; single-COG and tile-envelope
paths are unchanged.  CogCommitMessage.paths carries written mini-COG
paths for Task 3 VRT build; pending_paths is always empty in mosaic mode.

21 new tests: grid coverage, pixel fidelity, pruneEmpty, overlapPercent
halo, pickle round-trip, non-divisible dims, single-COG regression.

Co-authored-by: Isaac
commit() in mosaic mode now calls _build_mosaic_vrt() when write_vrt=True
(the default). The helper is pure Python + rasterio (no osgeo/GDAL bindings),
compatible with the light tier on Serverless.

Implementation details:
- Opens each mini-COG member with rasterio to collect transform / dims / CRS
  / dtype / nodata.
- Computes the mosaic bounding box from member extents; derives rasterXSize,
  rasterYSize, and GeoTransform via round() (float-safe for native-tiled grids).
- Emits a VRTDataset XML with one VRTRasterBand per source band, each
  containing one SimpleSource per tile (SrcRect always full tile, DstRect
  offset = pixel position within the mosaic).
- vrt_paths="relative" (default): relativeToVRT="1" + bare filename ->
  portable, moveable alongside the tiles.
- vrt_paths="absolute": full path + relativeToVRT="0".
- write_vrt=False: skips the VRT entirely (mosaic_opts=None single-COG path
  is unchanged).

Tests (test_mosaic_vrt.py, 11 new):
  VRT created; relative paths by default; width/height/CRS/transform match
  source; windowed read == source pixels; write_vrt=False; absolute paths;
  only written (non-pruned) tiles referenced; single-COG commit() regression;
  multi-band VRTRasterBand count.

Co-authored-by: Isaac
Add _mosaic.mint_vrt(tile_paths, out=None) — the Phase A "mint" face of the
mosaic API.  Builds a GDAL VRT over any dynamic tile set (filtered query
result, ad-hoc list) and returns the path to a transient file the caller
opens and discards.

Design:
- Reuses _build_mosaic_vrt (Task 3, pure xml.etree + rasterio) via a
  tempfile.mkdtemp() holding-dir; vrt_paths="absolute" so member paths
  resolve regardless of where the transient VRT lives.
- out= moves the VRT to a caller-supplied destination (temp dir cleaned up).
- No osgeo / Spark session / _jvm — light-tier / Serverless safe.

Tests (python/geobrix/test/ds/test_mosaic_mint.py):
  6 TDD tests: open-as-mosaic, windowed-read, dynamic-subset,
  transient-location, out-path, no-osgeo; 11+21 regression tests pass.

Co-authored-by: Isaac
Detects a .vrt path in RasterGbxReader.partitions() and expands it into
one whole-file virtual tile per member mini-COG instead of treating the
VRT as a single opaque source.  Member paths are extracted by parsing
the VRT XML (SourceFilename elements; relativeToVRT="1" resolved
relative to the VRT directory, absolute paths used as-is) using
defusedxml (with stdlib fallback).  Multi-band VRTs reference each tile
once per band; deduplication preserves first-occurrence order.

Reuses the existing whole-file virtual-tile emission path entirely —
_plan_partitions_for_file with emit_virtual=True, window=None.  All
existing AOI/split options compose unchanged.  CogGbxReader inherits the
behaviour for free (no code change in cog.py); both raster_gbx and
cog_gbx now expand .vrt, giving the write→read round-trip.

Load convention: point at the .vrt explicitly to trigger expansion.
Directory walk (pointing at a containing directory) is unchanged and
still includes .vrt files as rasterio sources.  Connect-safe: member
enumeration is pure Python / stdlib I/O with no SparkSession, no _jvm.

Tests: test_vrt_read_expand.py (6 tests — count, whole-file-virtual
struct, rst_avg unchanged proof, minted-VRT absolute-paths, cog_gbx
round-trip, no-osgeo lint); all pass.  No regression in
test_raster_datasource, test_reader_layout, test_raster_virtual,
test_cog_reader, test_mosaic_vrt, test_mosaic_mint, test_mosaic_write.

Co-authored-by: Isaac
Task 6 of the Phase-A native mini-COG + VRT mosaic SDD.

Adds python/geobrix/test/ds/test_mosaic_roundtrip.py with three
end-to-end integration tests that prove Tasks 1–5 compose correctly:

1. test_prepare_expand_rst_avg: cog_gbx mosaic write → raster_gbx VRT
   expansion → rst_avg. Asserts row count == member count, rst_avg values
   match direct rasterio means per tile, and member pixel means match the
   source region pixels (pixel-perfect write).

2. test_windowed_via_mint_vrt: mint_vrt over the mini-COGs → rasterio
   open → cross-tile windowed read. Asserts pixel-exact equality with the
   same window from the original source (straddles tile_0_0 / tile_0_1).

3. test_bbox_filtered_vrt_read: mosaic.vrt loaded with clipPolygons inset
   within tile_0_0's spatial extent → asserts exactly 1 row returned for
   the correct member.

No composition gaps found. All five existing mosaic suites are green.

Co-authored-by: Isaac
Fix-round for Phase A native mini-COG + VRT mosaic (light tier, pure Python).

FIX 1 (headline, silent data loss): _write_mosaic named tiles
tile_<row>_<col>.tif in a flat out_dir with no source discriminator, so
two source rasters in one write (or two partitions sharing out_dir)
collided -- cog_skip_if_exists skipped one / the other overwrote it,
losing pixels and leaving a VRT presenting one source as the whole
mosaic. Add _source_discriminator (stable per-source, single alnum token,
no underscore) and name tiles tile_<disc>_<row>_<col>.tif. The disc hashes
the resolved local path, so the same source is deterministic across
partitions (idempotent) while distinct sources never collide.

FIX 3 (tiled-source edge tiles): a tiled/COG source profile carries
tiled=True + blockxsize/blockysize (e.g. 512); a partial-edge tile smaller
than the block carried those into the intermediate GTiff write. Drop
tiled/blockxsize/blockysize/interleave before writing the window (the
final COG re-tiles regardless) -- correctness-hardening.

FIX 4 (overwrite skips stale VRT): overwrite cleanup globbed only *.<ext>,
stranding a stale mosaic.vrt. Also sweep *.vrt.

FIX 5 (dir-load double-counts VRT + members): the raster directory walk
read mosaic.vrt as one whole-mosaic source on top of the tile_*.tif it
indexes. Exclude *.vrt from _list_source_files (a .vrt is honored only
when the load path points directly at it).

Tests: two-source no-collision (+VRT references both), tiled-source
partial-edge tiles, overwrite removes stale vrt, dir-load excludes vrt,
plus updated discriminator-dependent naming assertions. All green in
Docker (gbx:test:python): 111 passed.

Co-authored-by: Isaac
_parse_vrt_members prefers defusedxml but falls back to stdlib xml.etree
when it is absent; defusedxml was in no extra, so a production light
install parsed untrusted external .vrt indexes with the vulnerable stdlib
parser (XXE / billion-laughs). Add defusedxml>=0.7,<1 to [light-base] so
[light], [light_dbr19], and the [*_all] umbrellas inherit it; keep the
import fallback for resilience.

Lock note: uv is present in the dev container but has no PyPI network, so
a full `uv pip compile` regeneration is not possible here. defusedxml
0.7.1 is dependency-free and pure Python, so the only correct change to
each hash-pinned lock is adding its two-hash block -- inserted by hand
using the repo-trusted hashes already checked in via
requirements-dev-container.txt. The two flat .in files list it
explicitly; the two -all .in files (.[light_all]/.[light_dbr19_all]) pick
it up from the [light-base] change on any future recompile. Follow-up
(non-blocking): re-run uv pip compile in a network-enabled environment to
confirm byte-parity (only expected delta is the defusedxml block).

Co-authored-by: Isaac
…Databricks Spatial

Page previously opened raster-only and mentioned vector only as an afterthought.
Rewrites the intro, the connect-aware cap section, the FILE fast path section, and
the Serverless-safe tables to cover raster and vector consistently throughout. Adds a
dedicated "Vector memory profile" section documenting bbox/where pushdown, GPKG/FileGDB
worker-local staging, and the FILE-column vector table fast path — all sourced from
pyvx/file_read.py and ds/vector.py. Clarifies that the connect-aware cap governs raster
tile materialization only; vector's memory is bounded by pushdown + FILE table, not by
that threshold. Corrects the writer note: driver commit streams one Arrow batch at a time
(not chunkSize-bounded, which is a reader-side option); GeoJSONL has no driver merge at
all; OpenFileGDB requires native GDAL / classic clusters. sidebars.js moves the page to
right after 'databricks-spatial'.

Co-authored-by: Isaac
…e-rasters/virtual-tiles

New page docs/docs/api/vrt-mosaic.mdx covers:
- Write: cog_gbx mosaic mode options (gridSystem, tileSize, overlapPercent,
  mergeStrategy, pruneEmpty, writeVrt, vrtPaths) with example
- mint_vrt: on-demand transient VRT API (parameters, returns, Connect-safe note)
- Read: raster_gbx/cog_gbx .vrt expansion → one virtual tile per member,
  directory vs VRT load distinction, clipPolygons spatial filtering
- Serverless note: windowed tile reads are bounded; source never fully in RAM
- Upcoming: grid-aligned mosaics (quadbin/BNG/H3) + windowed rendering

Cross-links added to: readers-writers (Next Steps), writers/cog (new VRT
mosaic section + Next steps), readers/cog (Next steps), api/large-rasters
(See also — 308 MB → 130-tile proof), api/virtual-tiles (See also).
sidebars.js: api/vrt-mosaic inserted after api/virtual-tiles in RasterX group.

Co-authored-by: Isaac
The Large Rasters "Memory footprint / standard Serverless" section discusses the
~1 GB per-task ceiling but didn't point at the authoritative Serverless & Memory
page. Add an inline cross-link to the connect-aware memory model (stream cap,
materialize-vs-virtual, FILE Delta-table fast path). Docs build clean.

Co-authored-by: Isaac
…ded vs DBLabs Mosaic)

The public option key "mosaic" on the cog_gbx writer collided with the
DBLabs Mosaic project that GeoBrix succeeds.  .option("mosaic", true) reads
as invoking the old project.  Rename to "vrtMosaic" — unambiguous, matches
the VRT-mosaic feature name — across the parser, all five test files, and the
vrt-mosaic / cog writer doc pages.  Beta has no aliases; old key is gone.

Co-authored-by: Isaac
The opening sentence framed a VRT mosaic as strictly a directory of
bounded mini-COG tiles. A VRT indexes whole COGs just as well as the
mini-COGs the writer produces; broaden the definition so users don't
read the mini-COG tiling as a precondition of the mosaic form.

Co-authored-by: Isaac
…stem-required

The COG writer page buried VRT-mosaic capability in a late section and
still carried a pre-rename `mosaic=true` in its prose; the reader page
only linked to VRT Mosaics without explaining that cog_gbx expands a
.vrt. Add a top-of-page VRT summary to both, a writer mosaic-mode options
table, and a reader "Reading a VRT mosaic" section spelling out the
options that drive a VRT load (.vrt recognition, clipPolygons+clipCrs
filtering, virtualTiles).

Also correct a misleading example: gridSystem defaults to "none"
(parse_mosaic_options, cog_writer.py:481) and mosaic mode is triggered by
vrtMosaic alone, so gridSystem is not required. Drop the redundant
gridSystem="none" option from the examples on both the VRT Mosaics and
COG writer pages and fix the option-table default from '—' to 'none'.

Co-authored-by: Isaac
…lume)

The mint_vrt example used out=/tmp/query_mosaic.vrt without explaining
that /tmp is driver-local and ephemeral, which read as a reachable
persist target. mint_vrt bakes ABSOLUTE member paths (_mosaic.py:88,
vrt_paths=absolute), so usability depends on both the .vrt file and its
member tiles resolving wherever it is opened.

Show both a driver-local (/tmp, transient one-shot read) and a
/Volumes (shared, durable, worker/external-client readable) example, add
a reachability note, and point users to the writer's relative-path
mosaic.vrt when they want a portable movable index instead.

Co-authored-by: Isaac
A mosaic.vrt need not originate from the cog_gbx writer — mint_vrt or any
GDAL tool can produce one. Loosen "written by" to "can be written by" so
the writer reads as one source of a VRT, not the only one.

Co-authored-by: Isaac
The VRT Mosaics page opened with prose only. Add a house-style left-to-right
hero diagram so the shipped native pattern is graspable at a glance and the
image is reusable in slides.

The generator emits hand-built SVG (Inter typography, shared stage-card chrome,
connector arrows), rasterized to PNG via Chrome headless at 2x and cropped with
PIL, matching the sibling rasterx diagrams. Four stages: Source (one large
raster too big for a single COG or per-task RAM) -> Tile (the cog_gbx writer in
mosaic mode reads window-by-window into bounded mini-COGs) -> Index (the
portable mosaic.vrt lens, persisted or minted on demand, openable in any GDAL
tool) -> Read (a reader expands it into one virtual tile per member for
distributed per-tile rst_* work). Reserved teal marks only the expanded virtual
member tiles; mini-COG files and stage chrome use the neutral accents.

Embedded at the top of the page with full-sentence descriptive alt text.

Co-authored-by: Isaac
serverless-and-memory.mdx sits at the docs/docs/ top level, so its two
../readers-writers links resolved above /docs/ to a 404. Its sibling
links already use ./api/... correctly; make the readers-writers links
./readers-writers to match. onBrokenLinks='warn' let these slip past the
build (see loose-ends note re: the build script's misleading success).

Co-authored-by: Isaac
onBrokenLinks='warn' means docusaurus build exits 0 even with broken
internal links, so the script's flat '✅ links resolve' banner was
misleading (it masked two real 404s on serverless-and-memory earlier).

Add --strict: capture build output, detect Docusaurus's broken-link
report, and exit non-zero when present — a green validation gate for
pre-push/CI. Default stays permissive (warn, exit 0) so the routine build
+ dev-server-restart flow isn't blocked by a stray link. Under --strict
the dev server is still restarted (port 3000 never left down); only the
exit code fails. The success banner now claims 'all links resolve' only
when Docusaurus reported none broken; otherwise it warns.

Co-authored-by: Isaac
mint_vrt's default temp-dir VRT was caller-cleanup. Add minted_vrt, a
contextmanager wrapping mint_vrt that removes the temp dir (and the .vrt
in it) on exit, including on exception. Member tiles are absolute-path
references, never copied, so cleanup never touches them. mint_vrt itself
is unchanged (additive). +2 tests (happy path + exception path); 8 passed.

Co-authored-by: Isaac
_build_mosaic_vrt applied the single dataset-level ds.nodata to every VRT
band. Switch to rasterio ds.nodatavals (one entry per band) so differing
per-band nodata is carried into the VRT; behavior-identical for the common
uniform case. Unknown dtype already falls back to Float32 (unchanged).
Adds a guard test (multiband uniform nodata gives NoDataValue on every band).

Co-authored-by: Isaac
The VRT Mosaics page showed three inline Python blocks (prepare, read/expand,
mint) that nothing executed, violating the repo rule that tests are the
documentation source. Move the code into vrt_mosaic_examples.py as display
constants (VRT_PREPARE / VRT_READ_EXPAND / VRT_MINT) plus executable functions
that run the real cog_gbx mosaic writer, raster_gbx VRT expansion, and mint_vrt
windowed read against sample data with assertions. The page now renders them via
CodeFromTest so the shown code is the tested code. Inline out= fragments and the
spatial-filtering snippet stay inline as illustrative fragments.

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant